Method and apparatus for deep neural network based inter-frame prediction in video coding

ABSTRACT

Video coding using neural network based inter-frame prediction is performed by generating a current reference frame by generating intermediate flows based on two input frames, performing backward warping of the input frames to generate reconstruction frames, and generating a fusion map and a residual map based on the input frames, the intermediate flows and the reconstruction frames. The video coding method further includes outputting an enhanced frame or a virtual reference picture by generating a feature map with different levels, based on the current reference frame, a first reference frame and a second reference frame, generating a predicted frame based on aligned features from the generated feature map by refining the current reference frame, the first reference frame, and the second reference frame, generating a final residual based on the predicted frame, and computing the enhanced frame as an output by adding the final residual to the current reference frame.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority to U.S. ProvisionalPatent Application No. 63/131,625, filed on Dec. 29, 2020, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

Uncompressed digital video can consist of a series of pictures, eachpicture having a spatial dimension of, for example, 1920×1080 luminancesamples and associated chrominance samples. The series of pictures canhave a fixed or variable picture rate (informally also known as framerate), of, for example 60 pictures per second or 60 Hz. Uncompressedvideo has significant bitrate requirements. For example, 1080p60 4:2:0video at 8 bit per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires close to 1.5 Gbit/s bandwidth. An hour of suchvideo requires more than 600 GByte of storage space.

Traditional video coding standards, such as the H.264/Advanced VideoCoding (H.264/AVC), High-Efficiency Video Coding (HEVC) and VersatileVideo Coding (VVC) share a similar (recursive) block-based hybridprediction/transform framework where individual coding tools like theintra/inter prediction, integer transforms, and context-adaptive entropycoding, are intensively handcrafted to optimize the overall efficiency.

SUMMARY

According to embodiments, a method of video coding using neural networkbased inter-frame prediction is performed by at least one processor andincludes generating intermediate flows based on input frames, generatingreconstruction frames by performing backward warping of the input frameswith the intermediate flows, generating a fusion map and a residual map,based on the input frames, the intermediate flows, and thereconstruction frames, generating a feature map with a plurality oflevels using a first neural network, based on a current reference frame,a first reference frame and a second reference frame, generating apredicted frame based on aligned features from the generated feature mapby refining the current reference frame, the first reference frame andthe second reference frame, generating a final residual based on thepredicted frame, and computing an enhanced frame as an output by addingthe final residual to the current reference frame.

According to embodiments, an apparatus for video coding using neuralnetwork based inter-frame prediction including at least one memoryconfigured to store program code and at least one processor configuredto read the program code and operate as instructed by the program code.The program code including first generating code configured to cause theat least one processor to generate intermediate flows based on inputframes, second generating code configured to cause the at least oneprocessor to perform backward warping of the input frames with theintermediate flows to generate reconstruction frames, fusion codeconfigured to cause the at least one processor to generate a fusion mapand a residual map, based on the input frames, the intermediate flowsand the reconstruction frames, third generating code configured to causethe at least one processor to generate a feature map with a plurality oflevels using a first neural network, based on a current reference frame,a first reference frame and a second reference frame, predicting codeconfigured to cause the at least one processor to predict a frame basedon aligned features from the generated feature map by refining thecurrent reference frame, the first reference frame and the secondreference frame, residual code configured to cause the at least oneprocessor to generate a final residual based on the predicted frame, andfourth generating code configured to cause the at least one processor togenerate an enhanced frame as an output by adding the final residual tothe current reference frame.

According to embodiments, a non-transitory computer-readable mediumstoring instructions that, when executed by at least one processor forvideo coding using neural network based inter-frame prediction, causethe at least one processor to generate intermediate flows based on inputframes, perform backward warping of the input frames with theintermediate flows to generate reconstruction frames, generate a fusionmap and a residual map based on the input frames, the intermediate flowsand the reconstruction frames, generate a feature map with a pluralityof levels using a first neural network, based on a current referenceframe, a first reference frame and a second reference frame, predict aframe based on aligned features from the generated feature map byrefining the current reference frame, the first reference frame and thesecond reference frame, generate a final residual based on the predictedframe, and generate an enhanced frame as an output by adding the finalresidual to the current reference frame.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an environment in which methods, apparatuses andsystems described herein may be implemented, according to embodiments.

FIG. 2 is a block diagram of example components of one or more devicesof FIG. 1.

FIG. 3 is a schematic of an example of virtual reference picturegeneration and insertion into a reference picture list.

FIG. 4 is a block diagram of a test apparatus of a virtual referencegeneration process, during a test stage, according to embodiments.

FIG. 5 is a detailed block diagram of an Optical Flow Estimation andIntermediate frame synthesizing module from the test apparatus of FIG.4, during a test stage, according to embodiments.

FIG. 6 is a detailed block diagram of a Detail Enhancement module fromthe test apparatus of FIG. 4, during a test stage, according toembodiments.

FIG. 7 is a detailed block diagram of a PCD Alignment module, during atest stage, according to embodiments.

FIG. 8 is a detailed block diagram of a TSA Fusion module, during a teststage, according to embodiments.

FIG. 9 is a detailed block diagram of the TSA Fusion module, during atest stage, according to another embodiment.

FIG. 10 is a flowchart of a method of video coding using neural networkbased inter-frame prediction, according to embodiments.

FIG. 11 is a block diagram of an apparatus of video coding using neuralnetwork based inter-frame prediction, according to embodiments.

DETAILED DESCRIPTION

This disclosure describes a Deep Neural Network (DNN)-based modelrelating to video coding and decoding. More specifically, a (DNN)-basedmodel that uses and generates virtual reference data from adjacentreference frames for inter-frame prediction as a Video FrameInterpolation (VFI) task and a detail enhancement module to furtherimprove the frame's quality and reduce artifacts (such as noises, blur,blocky effects, etc.) on motion boundaries.

One purpose of video coding and decoding can be the reduction ofredundancy in the input video signal, through compression. Compressioncan help reducing aforementioned bandwidth or storage spacerequirements, in some cases by two orders of magnitude or more. Bothlossless and lossy compression, as well as a combination thereof can beemployed. Lossless compression refers to techniques in which an exactcopy of the original signal can be reconstructed from the compressedoriginal signal. When using lossy compression, the reconstructed signalmay not be identical to the original signal, but the distortion betweenoriginal and reconstructed signal is small enough to make thereconstructed signal useful for the intended application. In the case ofvideo, lossy compression is widely employed. The amount of distortiontolerated depends on the application; for example, users of certainconsumer streaming applications may tolerate higher distortion thanusers of television contribution applications. The compression ratioachievable can reflect that: higher allowable/tolerable distortion canyield higher compression ratios.

Basically, the Spatiotemporal pixel neighborhoods are leveraged forpredictive signal construction, to obtain corresponding residuals forsubsequent transform, quantization, and entropy coding. On the otherhand, the nature of Deep Neural Networks (DNN) is to extract differentlevels of spatiotemporal stimuli by analyzing spatiotemporal informationfrom the receptive field of neighboring pixels. The capability ofexploring highly nonlinearity and nonlocal spatiotemporal correlationsprovide promising opportunity for largely improved compression quality.

One caveat of leveraging information from multiple neighboring videoframes is the complex motion caused by moving camera and dynamic scenes.Traditional block-based motion vectors cannot work well fornon-translational motions. Learning based optical flow methods canprovide accurate motion information at pixel-level, which is,unfortunately prone to error, especially along the boundary of movingobjects. This disclosure proposes to using a DNN-based model toimplicitly handle arbitrary complex motion in a data-driven fashion,without explicit motion estimation.

FIG. 1 is a diagram of an environment 100 in which methods, apparatusesand systems described herein may be implemented, according toembodiments.

As shown in FIG. 1, the environment 100 may include a user device 110, aplatform 120, and a network 130. Devices of the environment 100 mayinterconnect via wired connections, wireless connections, or acombination of wired and wireless connections.

The user device 110 includes one or more devices capable of receiving,generating, storing, processing, and/or providing information associatedwith platform 120. For example, the user device 110 may include acomputing device (e.g., a desktop computer, a laptop computer, a tabletcomputer, a handheld computer, a smart speaker, a server, etc.), amobile phone (e.g., a smart phone, a radiotelephone, etc.), a wearabledevice (e.g., a pair of smart glasses or a smart watch), or a similardevice. In some implementations, the user device 110 may receiveinformation from and/or transmit information to the platform 120.

The platform 120 includes one or more devices as described elsewhereherein. In some implementations, the platform 120 may include a cloudserver or a group of cloud servers. In some implementations, theplatform 120 may be designed to be modular such that software componentsmay be swapped in or out. As such, the platform 120 may be easily and/orquickly reconfigured for different uses.

In some implementations, as shown, the platform 120 may be hosted in acloud computing environment 122. Notably, while implementationsdescribed herein describe the platform 120 as being hosted in the cloudcomputing environment 122, in some implementations, the platform 120 maynot be cloud-based (i.e., may be implemented outside of a cloudcomputing environment) or may be partially cloud-based.

The cloud computing environment 122 includes an environment that hoststhe platform 120. The cloud computing environment 122 may providecomputation, software, data access, storage, etc. services that do notrequire end-user (e.g., the user device 110) knowledge of a physicallocation and configuration of system(s) and/or device(s) that hosts theplatform 120. As shown, the cloud computing environment 122 may includea group of computing resources 124 (referred to collectively as“computing resources 124” and individually as “computing resource 124”).

The computing resource 124 includes one or more personal computers,workstation computers, server devices, or other types of computationand/or communication devices. In some implementations, the computingresource 124 may host the platform 120. The cloud resources may includecompute instances executing in the computing resource 124, storagedevices provided in the computing resource 124, data transfer devicesprovided by the computing resource 124, etc. In some implementations,the computing resource 124 may communicate with other computingresources 124 via wired connections, wireless connections, or acombination of wired and wireless connections.

As further shown in FIG. 1, the computing resource 124 includes a groupof cloud resources, such as one or more applications (“APPs”) 124-1, oneor more virtual machines (“VMs”) 124-2, virtualized storage (“VSs”)124-3, one or more hypervisors (“HYPs”) 124-4, or the like.

The application 124-1 includes one or more software applications thatmay be provided to or accessed by the user device 110 and/or theplatform 120. The application 124-1 may eliminate a need to install andexecute the software applications on the user device 110. For example,the application 124-1 may include software associated with the platform120 and/or any other software capable of being provided via the cloudcomputing environment 122. In some implementations, one application124-1 may send/receive information to/from one or more otherapplications 124-1, via the virtual machine 124-2.

The virtual machine 124-2 includes a software implementation of amachine (e.g., a computer) that executes programs like a physicalmachine. The virtual machine 124-2 may be either a system virtualmachine or a process virtual machine, depending upon use and degree ofcorrespondence to any real machine by the virtual machine 124-2. Asystem virtual machine may provide a complete system platform thatsupports execution of a complete operating system (“OS”). A processvirtual machine may execute a single program, and may support a singleprocess. In some implementations, the virtual machine 124-2 may executeon behalf of a user (e.g., the user device 110), and may manageinfrastructure of the cloud computing environment 122, such as datamanagement, synchronization, or long-duration data transfers.

The virtualized storage 124-3 includes one or more storage systemsand/or one or more devices that use virtualization techniques within thestorage systems or devices of the computing resource 124. In someimplementations, within the context of a storage system, types ofvirtualizations may include block virtualization and filevirtualization. Block virtualization may refer to abstraction (orseparation) of logical storage from physical storage so that the storagesystem may be accessed without regard to physical storage orheterogeneous structure. The separation may permit administrators of thestorage system flexibility in how the administrators manage storage forend users. File virtualization may eliminate dependencies between dataaccessed at a file level and a location where files are physicallystored. This may enable optimization of storage use, serverconsolidation, and/or performance of non-disruptive file migrations.

The hypervisor 124-4 may provide hardware virtualization techniques thatallow multiple operating systems (e.g., “guest operating systems”) toexecute concurrently on a host computer, such as the computing resource124. The hypervisor 124-4 may present a virtual operating platform tothe guest operating systems, and may manage the execution of the guestoperating systems. Multiple instances of a variety of operating systemsmay share virtualized hardware resources.

The network 130 includes one or more wired and/or wireless networks. Forexample, the network 130 may include a cellular network (e.g., a fifthgeneration (5G) network, a long-term evolution (LTE) network, a thirdgeneration (3G) network, a code division multiple access (CDMA) network,etc.), a public land mobile network (PLMN), a local area network (LAN),a wide area network (WAN), a metropolitan area network (MAN), atelephone network (e.g., the Public Switched Telephone Network (PSTN)),a private network, an ad hoc network, an intranet, the Internet, a fiberoptic-based network, or the like, and/or a combination of these or othertypes of networks.

The number and arrangement of devices and networks shown in FIG. 1 areprovided as an example. In practice, there may be additional devicesand/or networks, fewer devices and/or networks, different devices and/ornetworks, or differently arranged devices and/or networks than thoseshown in FIG. 1. Furthermore, two or more devices shown in FIG. 1 may beimplemented within a single device, or a single device shown in FIG. 1may be implemented as multiple, distributed devices. Additionally, oralternatively, a set of devices (e.g., one or more devices) of theenvironment 100 may perform one or more functions described as beingperformed by another set of devices of the environment 100.

FIG. 2 is a block diagram of example components of one or more devicesof FIG. 1.

A device 200 may correspond to the user device 110 and/or the platform120. As shown in FIG. 2, the device 200 may include a bus 210, aprocessor 220, a memory 230, a storage component 240, an input component250, an output component 260, and a communication interface 270.

The bus 210 includes a component that permits communication among thecomponents of the device 200. The processor 220 is implemented inhardware, firmware, or a combination of hardware and software. Theprocessor 220 is a central processing unit (CPU), a graphics processingunit (GPU), an accelerated processing unit (APU), a microprocessor, amicrocontroller, a digital signal processor (DSP), a field-programmablegate array (FPGA), an application-specific integrated circuit (ASIC), oranother type of processing component. In some implementations, theprocessor 220 includes one or more processors capable of beingprogrammed to perform a function. The memory 230 includes a randomaccess memory (RAM), a read only memory (ROM), and/or another type ofdynamic or static storage device (e.g., a flash memory, a magneticmemory, and/or an optical memory) that stores information and/orinstructions for use by the processor 220.

The storage component 240 stores information and/or software related tothe operation and use of the device 200. For example, the storagecomponent 240 may include a hard disk (e.g., a magnetic disk, an opticaldisk, a magneto-optic disk, and/or a solid state disk), a compact disc(CD), a digital versatile disc (DVD), a floppy disk, a cartridge, amagnetic tape, and/or another type of non-transitory computer-readablemedium, along with a corresponding drive.

The input component 250 includes a component that permits the device 200to receive information, such as via user input (e.g., a touch screendisplay, a keyboard, a keypad, a mouse, a button, a switch, and/or amicrophone). Additionally, or alternatively, the input component 250 mayinclude a sensor for sensing information (e.g., a global positioningsystem (GPS) component, an accelerometer, a gyroscope, and/or anactuator). The output component 260 includes a component that providesoutput information from the device 200 (e.g., a display, a speaker,and/or one or more light-emitting diodes (LEDs)).

The communication interface 270 includes a transceiver-like component(e.g., a transceiver and/or a separate receiver and transmitter) thatenables the device 200 to communicate with other devices, such as via awired connection, a wireless connection, or a combination of wired andwireless connections. The communication interface 270 may permit thedevice 200 to receive information from another device and/or provideinformation to another device. For example, the communication interface270 may include an Ethernet interface, an optical interface, a coaxialinterface, an infrared interface, a radio frequency (RF) interface, auniversal serial bus (USB) interface, a Wi-Fi interface, a cellularnetwork interface, or the like.

The device 200 may perform one or more processes described herein. Thedevice 200 may perform these processes in response to the processor 220executing software instructions stored by a non-transitorycomputer-readable medium, such as the memory 230 and/or the storagecomponent 240. A computer-readable medium is defined herein as anon-transitory memory device. A memory device includes memory spacewithin a single physical storage device or memory space spread acrossmultiple physical storage devices.

Software instructions may be read into the memory 230 and/or the storagecomponent 240 from another computer-readable medium or from anotherdevice via the communication interface 270. When executed, softwareinstructions stored in the memory 230 and/or the storage component 240may cause the processor 220 to perform one or more processes describedherein. Additionally, or alternatively, hardwired circuitry may be usedin place of or in combination with software instructions to perform oneor more processes described herein. Thus, implementations describedherein are not limited to any specific combination of hardware circuitryand software.

The number and arrangement of components shown in FIG. 2 are provided asan example. In practice, the device 200 may include additionalcomponents, fewer components, different components, or differentlyarranged components than those shown in FIG. 2. Additionally, oralternatively, a set of components (e.g., one or more components) of thedevice 200 may perform one or more functions described as beingperformed by another set of components of the device 200.

A typical video compression framework can be described as follows. Givenan input video x including a plurality of image frames x₁, . . . ,x_(T), in the first motion estimation step, the frames are partitionedinto spatial blocks. Each block can be partitioned into smaller blocksiteratively and a set of motion vectors m_(t) between a current framex_(t) and a set of previous reconstructed frames {{circumflex over(x)}_(j)}_(t−1) is computed for each block. Note that the subscript tdenotes the current t-th encoding cycle, which may not match the timestamp of the image frame. Also, the set of previous reconstructed frames{{circumflex over (x)}_(j)}_(t−1) contain frames from multiple previousencoding cycles. Then, in the second motion compensation step, apredicted frame {tilde over (x)}_(t) is obtained by copying thecorresponding pixels of the set of previous reconstructed frames{{circumflex over (x)}_(j)}_(t−1) based on the motion vectors m_(t), anda residual r_(t) between the original frame x_(t) and the predictedframe {tilde over (x)}_(t) can be obtained (i.e. r_(t)=x_(t)−{tilde over(x)}_(t)). In the third motion compensation step, the residual r_(t) isquantized. The quantization step gives a quantized ŷ_(t). Both themotion vectors m_(t) and the quantized ŷ_(t) are encoded into bit steamsby entropy coding, which are sent to decoders. Then, on the decoderside, the quantized ŷ_(t) is dequantized (typically through inversetransformation like IDCT with the dequantized coefficients) to obtain arecovered residual {circumflex over (r)}_(t). Then the recoveredresidual {circumflex over (r)}_(t) is added back to the predicted frame{tilde over (x)}_(t) to obtain a reconstructed frame {circumflex over(x)}_(t) (i.e. {circumflex over (x)}_(t)={tilde over(x)}_(t)+{circumflex over (r)}_(t)). Additional components are furtherused to improve the visual quality of the reconstructed frame{circumflex over (x)}_(t). Typically, one or multiple of the followingenhancement modules can be selected to process the reconstructed frame{circumflex over (x)}_(t), including Deblocking Filter (DF),Sample-Adaptive Offset (SAO), Adaptive Loop Filter (ALF), etc.

In HEVC, VVC or other video coding frameworks or standards, the decodedpictures may be included in the reference picture list (RPL) and may beused for motion-compensated prediction as a reference picture and otherparameter prediction for coding the following picture(s) in the encodingor decoding order or may be used for intra-prediction or intra blockcopy for coding different regions or blocks of the current picture.

In embodiments, one or more virtual references may be generated andincluded in the RPL in both the encoder and decoder, or only in thedecoder. The virtual reference picture may be generated by one or moreprocesses including signal-processing, spatial or temporal filtering,scaling, weighted averaging, up-/down-sampling, pooling, recursiveprocessing with memory, linear system processing, non-linear systemprocessing, neural-network processing, deep-learning based processing,AI-processing, pre-trained network processing, machine-learning basedprocessing, on-line training network processing or their combinations.For the processing to generate the virtual reference(s), zero or moreforward reference pictures, which precede the current picture in bothoutput/display order and encoding/decoding order, and zero or morebackward reference pictures, which follow the current picture both inoutput/display order but precede the current picture inencoding/decoding order are used as input data. The output of theprocessing is the virtual/generated picture to be used as a newreference picture.

A DNN pre-trained network processing for virtual reference picturegeneration is described according to embodiments. FIG. 3 illustrates anexample of virtual reference picture generation and insertion into areference picture list including a hierarchical GOP structure 300, areference picture list 310, and a virtual reference generation process320.

As shown in FIG. 3, given the hierarchical GOP structure 300, when thecurrent picture has a picture order count (POC) equal to 3, usually, thedecoded picture with POC equal to 0, 2, 4 or 8 may be stored in adecoded picture buffer and some of them are included in the referencepicture list for decoding the current picture. As an example, thenearest decoded pictures with POC equal to 2 or 4 may be fed into thevirtual reference generation process 320 as input data. The virtualreference picture may be generated through one or multiple processes.The generated virtual reference picture may be stored in the decodedpicture buffer and included into the reference picture list 310 of thecurrent picture or one or more future pictures in decoding order. If thevirtual reference picture is included into the reference picture list310 of the current picture, the generated pixel data of the virtualreference picture may be used for motion compensated prediction asreference data, when it is indicated by a reference index that thevirtual reference picture is used.

In the same or another embodiment, the entire virtual referencegeneration process may include one of more signaling processing moduleswith one or more pre-trained neural network model or any pre-definedparameters. For example, the entire virtual reference generation process320 may be composed of a flow estimation module 330, a flow compensationmodule 340, and a detail enhancement module 350, as shown in FIG. 3. Theflow estimation module 330 may be an optical feature flow estimationprocess with DNN modeling. The flow compensation module 340 may be anoptical flow compensation and coarse intermediate frame synthesizingprocess with DNN modeling. The detail enhancement module 350 may be anenhancing process with DNN modeling.

A method and an apparatus of a DNN model for Video Frame Interpolation(VFI) according to embodiments will now be described in detail.

FIG. 4 is a block diagram of a test apparatus for a virtual referencegeneration process 400, during a test stage, according to embodiments.

As shown in FIG. 4, the virtual reference generation process 400includes an Optical Flow Estimation and Intermediate Frame synthesizingmodule 410 and a Detail Enhancement module 420.

In a random access configuration of a VVC decoder, two reference framesare fed into the Optical Flow Estimation and Intermediate Framesynthesizing module 410 to generate flow maps and then generate a coarseintermediate frame to be fed into the detail enhancement module 420along with the forward/backward reference frame to further improve theframe's quality. A more detailed description of the DNN modules, theOptical Flow Estimation and Intermediate Frame Synthesizing module 410and the Detail Enhancement module 420, used to perform the referenceframe generation process will be detailed later with reference to FIG. 5and FIG. 6, respectively.

Depending on the prediction structure and/or encoding configuration, twoor more different models may be trained and used, selectively. In therandom access configuration, which may use hierarchical predictionstructure with both forward and backward prediction reference picturesfor motion compensated prediction, one or more forward reference picturethat precedes the current picture in output (or display) order and oneor more backward reference picture that follows the current picture inoutput (or display) order are the inputs of the network. In the lowdelay configuration, which may have forward prediction referencepictures only for motion compensated prediction, two or more forwardreference pictures are the inputs of the network. For eachconfiguration, one or more different network models may be selected andused for inter-prediction.

Once one or more reference picture lists (RPLs) are constructed beforethe network inference is processed, the presence of the appropriatereference pictures that may be used as inputs of the network inferenceis checked in both encoder and decoder. If a backward reference picturethat has the same POC distance from the current picture as the POCdistance of a forward reference picture is present, the trained modelsfor random access configuration may be selected and used in networkinference to generate the intermediate (virtual) reference picture. Ifan appropriate backward reference picture is not present, two forwardreference pictures, one of which has a POC distance that is double theother, may be selected and used in network inference.

When the network model for inter-prediction explicitly uses an encodingconfiguration, one or more syntax elements in a high level syntaxstructure, such as parameter sets or headers, may indicate which modelsare used for the current sequence, picture, or slice. In an embodiment,all available network models in the current coded video sequence arelisted in a sequence parameter set (SPS), and the selected model foreach coded picture or slice is indicated by one or more syntax elementsin a picture parameter set (PPS) or picture/slice header.

The network topology and parameters may be explicitly specified in thespecification document of video coding standards, such as VVC, HEVC orAV1. In this case, the predefined network model may be used for theentire coded video sequence. If two or more network models are definedin the specification, one or more of them may be selected for each codedvideo sequence, coded picture or slice.

In a case where the customized network is used for each coded videosequence, picture or slice, the network topology and parameters may beexplicitly signaled in the high level syntax structure, such as aparameter set or SEI message in an elementary bitstream, or in ametadata track in the file format. In an embodiment, one or more networkmodels with network topologies and parameters are specified in one ormore SEI messages, where those SEI messages may be inserted into thecoded video sequence with different activation scopes. Further, thefollowing SEI message in the coded video bitstream may update thepreviously activated SEI message.

FIG. 5 is a detailed block diagram of the Optical Flow Estimation andIntermediate Frame Synthesizing module 410, during a test stage,according to embodiments.

As shown in FIG. 5, the Optical Flow Estimation and Intermediate FrameSynthesizing module 410 includes a Flow Estimation module 510, aBackward Warping module 520, and a Fusion Process module 530.

Two reference frames {I₀, I₁} are used as input to the Flow Estimationmodule 510 which approximates the intermediate flows {F_(0→t), F_(1→t)}from the perspective of the frame I_(t) that is expected to besynthesized. The Flow Estimation module 510 adopts a coarse-to-finestrategy with progressively increased resolutions: it iterativelyupdates a flow field. Conceptually, according to the iteratively updatedflow fields, corresponding pixels are moved from two input frames to thesame location in a latent intermediate frame.

With the estimated intermediate flows {F_(0→t), F_(1→t)}, the BackwardWarping module 520 generates coarse reconstruction frames or warpedframes {I_(0→t), I_(1→t)} by performing backward warping on the inputframes {I₀, I₁}. Backward warping of the frames may be performed byinversely mapping and sampling pixels of the latent intermediate frameto the input frames {I₀, I₁} to generate the warped frames {I_(0→t),I_(1→t)}. The Fusion Process module 530 takes the input frames {I₀, I₁},the warped frames {I_(0→t), I_(1→t)}, and the estimated intermediateflows {F_(0→t), F_(1→t)} as inputs. The Fusion Process module 530estimates a fusion map along with another residual map. The fusion mapand residual map are estimated maps that map feature variations in theinput. Then, the warped frames are linearly combined according to thefusion result and added with the residual map to do reconstruction toget an intermediate frame (referred to as Reference Frame I_(t) in FIG.5).

FIG. 6 is a detailed block diagram of the Detail Enhancement module 420,during a test stage, according to embodiments.

As shown in FIG. 6, the Detailed Enhancement module 420 includes a PCDAlignment module 610, a TSA Fusion module 620, and a Reconstructionmodule 630.

Assuming that reference frame I_(t) is a reference that all other frameswill be aligned to using and two reference frames {I_(t−1), I_(t+1)}together as input, the PCD (Pyramid, Cascading and Deformableconvolution) alignment module 610 refines the reference frame's(I_(t−1), I_(t), I_(t+1)) features as an aligned feature F_(t-aligned).Using the aligned feature F_(t-aligned), the TSA (Temporal and SpatialAttention) Fusion module 620 provides the weight of the feature map toapply attention to emphasize important features for subsequentrestoration and outputs a predicted frame I_(p). A more detaileddescription of the PCD Alignment module 610 and the TSA Fusion module620 will be detailed later with reference to FIG. 7 and FIG. 8,respectively.

Then, the Reconstruction module formats a final residual of thereference frame I_(t) based on the predicted frame I_(p). Finally, theAdd module 640 adds the formatted final residual to the reference frameI_(t) to get an enhanced frame I_(t-enhanced) as a final output of theDetailed Enhancement module 420.

FIG. 7 is a detailed block diagram of the PCD Alignment module 610inside the Detailed Enhancement module 420, during a test stage,according to embodiments. Modules in FIG. 7 having the same name andnumbering convention may be the same or one of a plurality of modulesperforming the described functions (e.g. Deformable Convolution module730).

As shown in FIG. 7, the PCD Alignment module 610 includes a FeatureExtraction module 710, an Offset Generation module 720, and a DeformableConvolution module 730. The PCD Alignment module 610 computes thealigned features F_(t-aligned) of the inter-frame.

First, using each reference frame in {I₀, I₁ . . . I_(t), I_(t+1) . . .} as input, the Feature Extraction module 710 computes a feature map{F₀, F₁ . . . F_(t), F_(t+1) . . . } with three different levels byusing a Feature Extraction DNN through forward inference. The differentlevels have different resolutions for feature compensation to capturedifferent levels of spatial/temporal information. For example, withlarger motion in the sequence, the smaller feature map will be able tohave a larger respective field and be able to handle a variety of objectoffset.

With different level feature maps, an Offset Generation module 720computes an offset map ΔP by concatenating the feature maps {F_(t),F_(t+1)} and then passing the concatenated feature maps through anoffset generation DNN. Then, the Deformable Convolution module 730calculates a new position of a DNN convolution kernel P_(new) by addingthe offset map ΔP to an original position P_(original) (i.e.ΔP+P_(original)) using a Temporal Deformable Convolution (TDC)operation. Note that the reference frame I_(t) can be any frame within{I₀, I₁ . . . I_(t), I_(t+1) . . . }. Without loss of generality, framesmay be ranked based on their time stamp in accenting order. In oneembodiment, when the current target is to enhance a currentreconstructed frame I′_(t), the reference frame I_(t) is used as theintermediate frame.

Since the new position P_(new) may be an irregular position and may notbe an integer, the TDC operation can be conducted by usinginterpolations (e.g., bilinear interpolation). By applying deformableconvolution kernel in the Deformable Convolution module 730,compensation features from different levels can be generated based onthe feature map of the level and the corresponding generated offsets.Deformable convolution kernel may then be applied to obtain thetop-level aligned feature F_(t-aligned) based on the offset map ΔP andone or more of the compensation features, by up-sampling and adding tothe upper-level compensation feature.

FIG. 8 is a detailed block diagram of the TSA Fusion module 620 insidethe Detailed Enhancement module 420, during a test stage, according toembodiments.

As shown in FIG. 8, the TSA Fusion module 620 includes an Activationmodule 810, an Element-wise Multiplication module 820, a FusionConvolution module 830, a Frame Reconstruction module 840, and a FrameSynthesis module 850. The TSA Fusion module 620 uses temporal andspatial attention. The goal of temporal attention is to compute framesimilarity in an embedding space. Intuitively, in an embedding space, aneighboring frame that is more similar to the reference frame should bepaid more attention to. Given the feature maps {F_(t−1), F_(t+1)} andaligned feature F_(t-aligned) of the center frame as input, theActivation module 810 uses a sigmoid activation function to restrict theinputs to [0, 1] as a simple convolution filters to get the temporalattention maps {F′_(t−1), F′_(t), F′_(t+1)} as an output of theActivation module 810. Note that for each spatial location, the temporalattention is spatial-specific.

Then, the Element-wise Multiplication module 820 multiplies the temporalattention maps {F′_(t−1), F′_(t), F′_(t+1)} in a pixel-wise manner withthe aligned feature F_(t-aligned). An extra fusion convolution layer isadopted in the Fusion Convolution module 830 to obtain aggregatedattention-modulated features F_(t-aligned-TSA). Using the temporalattention maps {F′_(t−1), F′_(t), F′_(t+1)} and the attention-modulatedfeatures F_(t-aligned-TSA), the Frame Reconstruction module 840 uses aFrame Reconstruction DNN through feed forward inference computation togenerate aligned frames I_(t-aligned). Then, the aligned framesI_(t-aligned) are passed through a Frame Synthesis module 850 togenerate the synthesized predicted frame I_(p) as a final output.

FIG. 9 is a detailed block diagram of the TSA Fusion module 620 insidethe Detailed Enhancement module 420, during a test stage, according toanother example embodiment. Modules in FIG. 9 having the same name andnumbering convention may be the same or one of a plurality of modulesperforming the described functions.

As shown in FIG. 9, the TSA Fusion module 620 according to thisembodiment includes an Activation module 910, an Element-wiseMultiplication module 920, a Fusion Convolution module 930, aDown-sampled Convolution (DSC) module 940, an Up-sampling and Add module950, a Frame Reconstruction module 960, and a Frame Synthesis module970. Similar to embodiments of FIG. 8, TSA Fusion module 620 usestemporal and spatial attention. Given the feature maps {F_(t−1),F_(t+1)} and aligned feature F_(t-aligned) of frame the center as input,the Activation module 910 uses a sigmoid activation function to restrictthe inputs to [0, 1] as a simple convolution filters to get the temporalattention maps {M_(t−1), M_(t), M_(t+1)} as an output of the Activationmodule 910.

Then, the Element-wise Multiplication module 920 multiplies the temporalattention maps {M_(t−1), M_(t), M_(t+1)} in a pixel-wise manner with thealigned feature F_(t-aligned). Fusion convolution is performed on theproduct, in the Fusion Convolution module 930, to generate fusedfeatures F_(fused). The fused features F_(fused) are passed down sampledand convolved by the DSC module 940. Another convolution layer may beadopted and processed by the DSC module 940, as shown in FIG. 9. Theoutput from each layer of the DSC module 940 is input to the Up-samplingand Add module 950. An extra layer of the up sampling and adding is beapplied along with the fused features F_(fused) as input to the extralayer, as shown in FIG. 9. The Up-sampling and Add modules 950 generatefused attention maps M_(t-fused). The Element-wise Multiplication module920 multiplies the fused attention maps M_(t-fused) in a pixel-wisemanner with the fused features F_(fused) to generate a TSA alignedfeature F_(t-TSA). Using the temporal attention maps {M_(t−1), M_(t),M_(t+1)} and the TSA aligned feature F_(t-TSA), the Frame Reconstructionmodule 960 uses a Frame Reconstruction DNN through feed forwardinference computation to generate TSA aligned frames I_(t-TSA). Then,the aligned frames I_(t-TSA) are passed through a Frame Synthesis module970 to generate the synthesized predicted frame I_(p) as a final output.

FIG. 10 is a flowchart of a method 1000 of video coding using neuralnetwork based inter-frame prediction, according to embodiments.

In some implementations, one or more process blocks of FIG. 10 may beperformed by the platform 120. In some implementations, one or moreprocess blocks of FIG. 10 may be performed by another device or a groupof devices separate from or including the platform 120, such as the userdevice 110.

As shown in FIG. 10, in operation 1001, the method 1000 includesgenerating intermediate flows based on input frames. The intermediateflows may further be iteratively updated and corresponding pixels movedfrom two input frames to the same location in a latent intermediateframe.

In operation 1002, the method 1000 includes generating reconstructionframes by performing backward warping of the input frames with theintermediate flows.

In operation 1003, the method 1000 includes generating a fusion map anda residual map, based on the input frames, the intermediate flows, andthe reconstruction frames.

In operation 1004, the method 1000 includes generating a feature mapwith a plurality of levels using a first neural network, based on acurrent reference frame, a first reference frame, and a second referenceframe. The current reference frame may be generated by linearlycombining the reconstruction frames according to the fusion map andadding the combined reconstruction frames with the residual map.Further, the first reference frame may be a reference frame thatprecedes the current reference frame in an output order and the secondreference frame may be a reference frame that follows the currentreference frame in the output order.

In operation 1005, the method 1000 includes generating a predicted framebased on aligned features from the generated feature map by refining thecurrent reference frame, the first reference frame, and the secondreference frame. Specifically, the predicted frame may be generated byfirst performing convolution to obtain attention maps, generatingattention features, based on the attention maps and the alignedfeatures, generating an aligned frame, based on the attention maps andthe attention features, and then synthesizing the aligned frame toobtain the predicted frame.

The aligned features may be generated by computing an offset for theplurality of levels of the feature map generated in operation 1004 andperforming deformable convolution to generate compensation features forthe plurality of levels. The aligned features may then be generatedbased on the offset and at least one of the generated compensationfeatures.

In operation 1006, the method 1000 includes generating a final residualbased on the predicted frame. A weight of the feature map, generated inoperation 1004, may also be generated, emphasizing important featuresfor generation of subsequent final residuals.

In operation 1007, the method 1000 includes computing an enhanced frameas an output by adding the final residual to the current referenceframe.

Although FIG. 10 shows example blocks of the method, in someimplementations, the method may include additional blocks, fewer blocks,different blocks, or differently arranged blocks than those depicted inFIG. 10. Additionally, or alternatively, two or more of the blocks ofthe method may performed in parallel.

FIG. 11 is a block diagram of an apparatus 1100 of video coding usingneural network based inter-frame prediction, according to embodiments.

As shown in FIG. 11, the apparatus includes first generating code 1101,second generating code 1102, fusion code configured 1103, thirdgenerating code 1104, predicting code 1105, residual code 1106, andfourth generating code 1107.

The first generating code 1101 is configured to cause the at least oneprocessor to generate intermediate flows based on input frames.

The second generating code 1102 is configured to cause the at least oneprocessor to perform backward warping of the input frames with theintermediate flows to generate reconstruction frames.

The fusion code configured 1103 is configured to cause the at least oneprocessor to generate a fusion map and a residual map, based on theinput frames, the intermediate flows, and the reconstruction frames.

The third generating code 1104 is configured to cause the at least oneprocessor to generate a feature map with a plurality of levels using afirst neural network, based on a current reference frame, a firstreference frame, and a second reference frame.

The predicting code 1105 is configured to cause the at least oneprocessor to predict a frame based on aligned features from thegenerated feature map by refining the current reference frame, the firstreference frame, and the second reference frame.

The residual code 1106 is configured to cause the at least one processorto generate a final residual based on the predicted frame.

The fourth generating code 1107 is configured to cause the at least oneprocessor to generate an enhanced frame as an output by adding the finalresidual to the current reference frame.

Although FIG. 11 shows example blocks of the apparatus, in someimplementations, the apparatus may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 11. Additionally, or alternatively, two or more of theblocks of the apparatus may be combined.

The apparatus may further include updating code configured to cause theat least one processor to iteratively update the intermediate flows andmove corresponding pixels from two input frames to the same location ina latent intermediate frame, reference frame code configured to causethe at least one processor to generate the current reference frame bylinearly combining the reconstruction frames according to the fusion mapand adding the combined reconstruction frames with the residual map,determining code configured to cause the at least one processor todetermine a weight of the feature map, wherein the weight emphasizesimportant features for generating subsequent final residuals, computingcode configured to cause the at least one processor to compute an offsetfor the plurality of levels, compensation feature generating codeconfigured to cause the at least one processor to perform deformableconvolution to generate compensation features for the plurality oflevels, aligned feature generating code configured to cause the at leastone processor to generate the aligned features based on the offset andat least one of the generated compensation features, performing codeconfigured to cause the at least one processor to perform convolution toobtain attention maps, attention feature generating code configured tocause the at least one processor to generate attention features based onthe attention maps and the aligned features, aligned frame generatingcode configured to cause the at least one processor to generate analigned frame, based on the attention maps and the attention features,using a second neural network, and synthesizing code configured to causethe at least one processor to synthesize the aligned frame to obtain thepredicted frame.

Comparing with traditional inter-frame generation approaches, theproposed method performs DNN-based network in video coding. Instead ofcomputing an explicit motion vector or motion flow that either cannothandle complex motion or is prone to error, the proposed methodsdirectly take data from a Reference Picture List (RPL) to generate avirtual reference frame. And then the enhanced Deformable Convolution(DCN) is applied to capture the offset of pixels and implicitlycompensate large complex motion for further detailed enhancement.Finally, a high-quality enhanced frame is reconstructed by a DNN model.

The proposed methods may be used separately or combined in any order.Further, each of the methods (or embodiments) may be implemented byprocessing circuitry (e.g., one or more processors or one or moreintegrated circuits). In one example, the one or more processors executea program that is stored in a non-transitory computer-readable medium.

The present disclosure provides illustration and description, but is notintended to be exhaustive or to limit the implementations to the preciseform disclosed. Modifications and variations are possible in light ofthe present disclosure or may be acquired from practice of theimplementations.

As used herein, the term component is intended to be broadly construedas hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, maybe implemented in different forms of hardware, firmware, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods were described herein without reference tospecific software code—it being understood that software and hardwaremay be designed to implement the systems and/or methods based on thedescription herein.

Even though combinations of features are recited in the claims and/ordisclosed in the specification, these combinations are not intended tolimit the disclosure of possible implementations. In fact, many of thesefeatures may be combined in ways not specifically recited in the claimsand/or disclosed in the specification. Although each dependent claimlisted below may directly depend on only one claim, the disclosure ofpossible implementations includes each dependent claim in combinationwith every other claim in the claim set.

No element, act, or instruction used herein may be construed as criticalor essential unless explicitly described as such. Also, as used herein,the articles “a” and “an” are intended to include one or more items, andmay be used interchangeably with “one or more.” Furthermore, as usedherein, the term “set” is intended to include one or more items (e.g.,related items, unrelated items, a combination of related and unrelateditems, etc.), and may be used interchangeably with “one or more.” Whereonly one item is intended, the term “one” or similar language is used.Also, as used herein, the terms “has,” “have,” “having,” or the like areintended to be open-ended terms. Further, the phrase “based on” isintended to mean “based, at least in part, on” unless explicitly statedotherwise.

What is claimed is:
 1. A method of video coding using neural network based inter-frame prediction, the method being performed by at least one processor, and the method comprising: generating intermediate flows based on input frames; generating reconstruction frames by performing backward warping of the input frames with the intermediate flows; generating a fusion map and a residual map, based on the input frames, the intermediate flows, and the reconstruction frames; generating a feature map with a plurality of levels using a first neural network, based on a current reference frame, a first reference frame, and a second reference frame; generating a predicted frame based on aligned features from the generated feature map by refining the current reference frame, the first reference frame, and the second reference frame; generating a final residual based on the predicted frame; and computing an enhanced frame as an output by adding the final residual to the current reference frame.
 2. The method of claim 1, wherein the intermediate flows are iteratively updated and corresponding pixels are moved from two input frames to the same location in a latent intermediate frame.
 3. The method of claim 1, further comprising generating the current reference frame by linearly combining the reconstruction frames according to the fusion map and adding the combined reconstruction frames with the residual map.
 4. The method of claim 1, wherein the first reference frame is a reference frame that precedes the current reference frame in an output order and the second reference frame is a reference frame that follows the current reference frame in the output order.
 5. The method of claim 1, further comprising determining a weight of features in the feature map, wherein the weight emphasizes a subset of features for generation of subsequent final residuals.
 6. The method of claim 1, further comprising: computing an offset for the plurality of levels; performing deformable convolution to generate compensation features for the plurality of levels; and generating the aligned features based on the offset and at least one of the generated compensation features.
 7. The method of claim 1, further comprising: performing convolution to obtain fused attention maps; generating attention features based on the attention maps and the aligned features; generating an aligned frame, based on the attention maps and the attention features, using a second neural network; and synthesizing the aligned frame to obtain the predicted frame.
 8. An apparatus for video coding using neural network based inter-frame prediction, the apparatus comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code, the program code comprising: first generating code configured to cause the at least one processor to generate intermediate flows based on input frames; second generating code configured to cause the at least one processor to perform backward warping of the input frames with the intermediate flows to generate reconstruction frames; fusion code configured to cause the at least one processor to generate a fusion map and a residual map, based on the input frames, the intermediate flows, and the reconstruction frames; third generating code configured to cause the at least one processor to generate a feature map with a plurality of levels using a first neural network, based on a current reference frame, a first reference frame, and a second reference frame; predicting code configured to cause the at least one processor to predict a frame based on aligned features from the generated feature map by refining the current reference frame, the first reference frame, and the second reference frame; residual code configured to cause the at least one processor to generate a final residual based on the predicted frame; and fourth generating code configured to cause the at least one processor to generate an enhanced frame as an output by adding the final residual to the current reference frame.
 9. The apparatus of claim 8, further comprising updating code configured to cause the at least one processor to iteratively update the intermediate flows and move corresponding pixels from two input frames to the same location in a latent intermediate frame.
 10. The apparatus of claim 8, further comprising reference frame code configured to cause the at least one processor to generate the current reference frame by linearly combining the reconstruction frames according to the fusion map and adding the combined reconstruction frames with the residual map.
 11. The apparatus of claim 8, wherein the first reference frame is a reference frame that precedes the current reference frame in an output order and the second reference frame is a reference frame that follows the current reference frame in the output order.
 12. The apparatus of claim 8, further comprising determining code configured to cause the at least one processor to determine a weight of features in the feature map, wherein the weight emphasizes a subset of features for generating subsequent final residuals.
 13. The apparatus of claim 8, further comprising: computing code configured to cause the at least one processor to compute an offset for the plurality of levels; compensation feature generating code configured to cause the at least one processor to perform deformable convolution to generate compensation features for the plurality of levels; and aligned feature generating code configured to cause the at least one processor to generate the aligned features based on the offset and at least one of the generated compensation features.
 14. The apparatus of claim 8, further comprising: performing code configured to cause the at least one processor to perform convolution to obtain fused attention maps; attention feature generating code configured to cause the at least one processor to generate attention features based on the attention maps and the aligned features; aligned frame generating code configured to cause the at least one processor to generate an aligned frame, based on the attention maps and the attention features, using a second neural network; and synthesizing code configured to cause the at least one processor to synthesize the aligned frame to obtain the predicted frame.
 15. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor for video coding using neural network based inter-frame prediction, cause the at least one processor to: generate intermediate flows based on input frames; perform backward warping of the input frames with the intermediate flows to generate reconstruction frames; generate a fusion map and a residual map, based on the input frames, the intermediate flows, and the reconstruction frames; generate a feature map with a plurality of levels using a first neural network, based on a current reference frame, a first reference frame, and a second reference frame; predict a frame based on aligned features from the generated feature map by refining the current reference frame, the first reference frame, and the second reference frame; generate a final residual based on the predicted frame; and generate an enhanced frame as an output by adding the final residual to the current reference frame.
 16. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to update the intermediate flows and move corresponding pixels from two input frames to the same location in a latent intermediate frame.
 17. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to generate the current reference frame by linearly combining the reconstruction frames according to the fusion map and adding the combined reconstruction frames with the residual map, and wherein the first reference frame is a reference frame that precedes the current reference frame in an output order and the second reference frame is a reference frame that follows the current reference frame in the output order.
 18. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to determine a weight of features in the feature map, wherein the weight emphasizes a subset of features for generating subsequent final residuals.
 19. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: compute an offset for the plurality of levels; perform deformable convolution to generate compensation features for the plurality of levels; and generate the aligned features based on the offset and at least one of the generated compensation features.
 20. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to: perform convolution to obtain fused attention maps; generate attention features based on the attention maps and the aligned features; generate an aligned frame, based on the attention maps and the attention features, using a second neural network; and synthesize the aligned frame to obtain the predicted frame. 